Depth map re-projection on user electronic devices

ABSTRACT

A method includes rendering, on displays of an extended reality (XR) display device, a first sequence of image frames based on image data received from an external electronic device associated with the XR display device. The method further includes detecting an interruption to the image data received from the external electronic device, and accessing a plurality of feature points from a depth map corresponding to the first sequence of image frames. The plurality of feature points includes movement and position information of one or more objects within the first sequence of image frames. The method further includes performing a re-warping to at least partially re-render the one or more objects based at least in part on the plurality of feature points and spatiotemporal data, and rendering a second sequence of image frames corresponding to the partial re-rendering of the one or more objects.

PRIORITY

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application No. 62/982,570, filed 27 Feb. 2020, whichis incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to depth maps, and, more particularly,to the re-projection of depth maps on user electronic devices.

BACKGROUND

An extended reality (XR) system may generally include acomputer-generated environment and/or a real-world environment thatincludes at least some XR artifacts. Such an XR system or world andassociated XR artifacts typically include various applications (e.g.,video games), which may allow users to utilize these XR artifacts bymanipulating their presence in the form of a computer-generatedrepresentation (e.g., avatar). In typical XR systems, image data may berendered on, for example, a lightweight, head-mounted display (HMD) thatmay be coupled through a physical wired connection to a base graphicsgeneration device responsible for generating the image data. In someinstances, it may be desirable to couple the HMD to the base graphicsgeneration device via a wireless network connection. However, certainwireless network connections may suffer reliability issues, causing theuser's XR experience to cease abruptly and without any precedingindication. It may be thus useful to provide techniques to improve XRsystems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example extended reality (XR) system.

FIG. 2A illustrates a detailed embodiment of an extended reality (XR)system with an available network connection.

FIG. 2B illustrates a detailed embodiment of an extended reality (XR)device with an unavailable network connection.

FIG. 2C illustrates another detailed embodiment of an extended reality(XR) system with an unavailable network connection.

FIG. 2D illustrates another detailed embodiment of an extended reality(XR) device with an unavailable network connection.

FIG. 3 illustrates is a flow diagram of a method for re-projecting depthmaps on user electronic devices.

FIG. 4 illustrates is a flow diagram of a method for providing depth mapfeature points for re-projecting depth maps on user electronic devices.

FIGS. 5A and 5B illustrate workflow diagram for determining one or morecurrent color frames and a frame extrapolation diagram, respectively.

FIGS. 6A, 6B, and 6C illustrate a workflow diagram for reducing featurepoints a workflow diagram for determining feature points, and a workflowdiagram for determining key point and depth sequence extrapolation,respectively.

FIGS. 7A, 7B, and 7C illustrate a workflow diagrams for determining andestimating head poses and object poses in the most recent image framesand/or derived image frames, respectively.

FIGS. 7D and 7E illustrate workflow diagrams and for determining objectpose estimation and performing a 2D image warping and re-projection,respectively.

FIG. 8 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The present embodiments are directed toward re-projecting depth maps onuser electronic devices. In particular embodiments, an extended reality(XR) electronic device may render, on one or more displays of the XRelectronic device, a first sequence of image frames based on image datareceived from a computing platform associated with the XR electronicdevice. The XR electronic device may then detecting an interruption tothe image data received from the computing platform associated with theXR display device. In particular embodiments, the XR electronic deviceand the computing platform may be communicatively connected to eachother via a wireless connection, in which the interruption is aninterruption to the wireless connection. In particular embodiments, thecomputing platform may access a number of feature points from a depthmap corresponding to the first sequence of image frames, in which thenumber of feature points may include movement and position informationof one or more objects within the first sequence of image frames.

In particular embodiments, prior to detecting the interruption to theimage data received from the computing platform, the XR electronicdevice may receive the number of feature points corresponding to thefirst sequence of image frames from the computing platform as abackground process. In particular embodiments, the XR electronic devicemay then store, to a memory of the XR electronic device, the number offeature points corresponding to the first sequence of image frames. Inparticular embodiments, the XR electronic device may perform are-warping process to at least partially re-render the one or moreobjects based at least in part on the plurality of feature points andspatiotemporal data. In particular embodiments, the XR electronic devicemay access current head pose data and predicted head pose data, in whichthe current head pose data and the predicted head pose data may beassociated with the number of feature points. In particular embodiments,the XR electronic device may also access current object pose data andpredicted object pose data, in which the current object pose data andthe predicted object pose data may be associated with the number offeature points.

In particular embodiments, the XR electronic device may perform there-warping process by determining one or more current color framescorresponding to the first sequence of image frames, and generating,based on the one or more current color frames, one or more updated colorframes corresponding to the first sequence of image frames. Inparticular embodiments, the XR electronic device may render, on the oneor more displays of the XR electronic device, a second sequence of imageframes corresponding to the partial re-rendering of the one or moreobjects. In this way, if a wireless network connection becomestemporarily unavailable, the second sequence of image frames may berendered for a predetermined period of time thereafter, or until secondimage data is received again from the computing platform. Thus, theuser's XR experience may not cease abruptly, and, instead, may graduallyand gracefully cease rendering in the case of network unavailability.

The present embodiments are further directed toward for providing depthmap feature points for re-projecting depth maps on user electronicdevices. In particular embodiments a computing platform may generateimage data corresponding to a first sequence of image frames. Inparticular embodiments, the computing platform may also access a depthmap corresponding to the first sequence of image frames. In particularembodiments, the depth map may include depth information for one or moremost recent image frames of the first sequence of image frames. Inparticular embodiments, the computing platform may also determine anumber of feature points from the depth map corresponding to the firstsequence of image frames based at least in part on a parametric datareduction (PDR) process, in which the number of feature points mayinclude movement and position information of one or more objects withinthe first sequence of image frames.

In particular embodiments, the computing platform may determine thenumber of feature points from the depth map may by selecting a subset offeature points of a total set of feature points included in the depthmap. In particular embodiments, the computing platform may alsodetermine the number of feature points from the depth map by determininga plurality of feature points within a predetermined viewing area. Inparticular embodiments, the computing platform may determine the numberof feature points within the predetermined viewing area by determining anumber of feature points within a predefined fovea display area. Inparticular embodiments, the number of feature points within thepredefined fovea display area may include a grouping of feature pointsbased at least part on a nearest-neighbor interpolation. In particularembodiments, the grouping of feature points may include a subgrouping offeature points grouped based at least part on a depth calculation. Inparticular embodiments, the computing platform may determine the numberof feature points by determining a pixel region corresponding to the oneor more objects within the first sequence of image frames, and dividingthe pixel region corresponding to the one or more objects into N pixelsubregions.

In particular embodiments, the computing platform may then extract anumber of feature points from the N pixel subregions, in which each ofthe number of feature points is extracted from a respective one of the Npixel subregions based on a confidence threshold. In particularembodiments, the computing platform may determine a position and anoptical flow for each of the plurality of feature points. In particularembodiments, the computing platform may then send the image data and thenumber of feature points to an XR electronic device that is external tothe electronic device. In particular embodiments, the computing platformand the XR display device may be communicatively connected to each othervia a wireless connection. In particular embodiments, the computingplatform may also provide current head pose data and predicted head posedata to the XR electronic device, in which the current head pose dataand the predicted head pose data may be associated with the number offeature points. In particular embodiments, the computing platform mayalso provide current object pose data and predicted object pose data tothe XR electronic device, in which the current object pose data and thepredicted object pose data are associated with the number of featurepoints. In this way, if a wireless network connection becomestemporarily unavailable, the second sequence of image frames may berendered for a predetermined period of time thereafter, or until secondimage data is received again from the computing platform. Thus, theuser's XR experience may not cease abruptly, and, instead, may graduallyand gracefully cease rendering in the case of network unavailability.

As used herein, “extended reality” may refer to a form ofelectronic-based reality that has been manipulated in some manner beforepresentation to a user, including, for example, virtual reality (VR),augmented reality (AR), mixed reality (MR), hybrid reality, simulatedreality, immersive reality, holography, or any combination thereof. Forexample, “extended reality” content may include completelycomputer-generated content or partially computer-generated contentcombined with captured content (e.g., real-world images). In someembodiments, the “extended reality” content may also include video,audio, haptic feedback, or some combination thereof, any of which may bepresented in a single channel or in multiple channels (such as stereovideo that produces a three-dimensional (3D) effect to the viewer).Further, as used herein, it should be appreciated that “extendedreality” may be associated with applications, products, accessories,services, or a combination thereof, that, for example, may be utilizedto create content in extended reality and/or utilized in (e.g., performactivities) in extended reality. Thus, “extended reality” content may beimplemented on various platforms, including a head-mounted device (HMD)connected to a host computer system, a standalone HMD, a mobile deviceor computing system, or any other hardware platform capable of providingextended reality content to one or more viewers.

FIG. 1 illustrates an example extended reality (XR) system 100, inaccordance with presently disclosed embodiments. In particularembodiments, the XR system 100 may include an XR electronic device 102,an input device 104, and a computing platform 106. In particularembodiments, a user may wear the XR electronic device 102 that maydisplay visual extended reality content to the user. The XR electronicdevice 102 may include an audio device that may provide audio extendedreality content to the user. In particular embodiments, the XRelectronic device 102 may include one or more cameras which can captureimages and videos of environments. The XR electronic device 102 mayinclude an eye tracking system to determine the vergence distance of theuser. In some embodiments, the XR electronic device 102 may include ahead-mounted display (HDM). The input device 104 may include, forexample, a trackpad and one or more buttons. The input device 104 mayreceive inputs from the user and relay the inputs to the computingplatform 106 and/or the XR electronic device 102. In particularembodiments, the XR electronic device 102 may be coupled to thecomputing platform 106 via one or more wireless networks 108. Inparticular embodiments, the computing platform 106 may include, forexample, a standalone host computing system, an on-board computer systemintegrated with the XR electronic device 102, a mobile device, or anyother hardware platform that may be capable of providing extendedreality content to and receiving inputs from the input device 104. Inparticular embodiments, the computing platform 106 may include, forexample, a cloud-based computing architecture (including one or moreservers 110 and data stores 112) suitable for hosting and servicing XRapplications or experiences executing on the XR electronic device 102.For example, in particular embodiments, the computing platform 106 mayinclude a Platform as a Service (PaaS) architecture, a Software as aService (SaaS) architecture, and an Infrastructure as a Service (IaaS),or other similar cloud-based computing architecture.

FIG. 2A illustrates a detailed embodiment of an extended reality (XR)system 200A for performing a 3D re-projection warping process, inaccordance with presently disclosed embodiments. As depicted, while awireless network connection is available, the computing platform 106Amay include a head pose tracking functional block 202A, a renderingengine 204A, a 3D re-projection warping functional block 206A, a keyfeature point and depth extraction functional block 208A, a head posefunctional block 210A, and object pose prediction functional block 212A.In particular embodiments, the computing platform 106 may generate imagedata corresponding to a first sequence of image frames via the renderingengine 204A. In particular embodiments, the computing platform 106A mayalso access a depth map corresponding to the first sequence of imageframes. In particular embodiments, the depth map may include depthinformation for one or more most recent image frames of the firstsequence of image frames. In particular embodiments, the computingplatform 106A may also determine, by the key feature point and depthextraction functional block 208A, a number of feature points from thedepth map corresponding to the first sequence of image frames based on aPDR process. In particular embodiments, the number of feature points mayinclude movement and position information (e.g., head pose data and headpose prediction data calculated by the head pose functional block 210A,object pose data and object pose prediction data by the object poseprediction functional block 212A) of one or more objects within thefirst sequence of image frames.

In particular embodiments, as further depicted by FIG. 2A, the computingplatform 106A may then send the image data and a number of featurepoints to the XR electronic device 102A. In particular embodiments, theXR electronic device 102A may include the storage functional block 214A,a latest inertial measurement unit (IMU) functional block 216, a latestIMU functional block 218, a 3D re-projection warping functional block220A, a data store 222, and a final re-projection and display functionalblock 224A. In one example, the latest IMU functional block 216 mayinclude IMU data captured at the time the head pose data and/or objectpose data is stored or calculated at the storage functional block 214A,such that the head pose data and/or object pose data may be impliedbased on the IMU data from the latest IMU functional block 216. In oneexample, the latest IMU functional block 218 may include real-time ornear real-time IMU data that may be recalculated (e.g., updated) beforethe first sequence of image frames are provided by the finalre-projection and display functional block 224A (e.g., when the networkconnection is still available) for rendering. The storage functionalblock 214A that may be utilized to receive and store the number offeature points corresponding to the first sequence of image frames fromthe computing platform 106A as a background process. In particularembodiments, the XR electronic device 102A may also receive current headpose data and predicted head pose data from the computing platform 106A.In particular embodiments, the XR electronic device 106A may alsoreceive current object pose data and predicted object pose data from thecomputing platform 106A. For example, in particular embodiments, thecurrent head pose data, predicted head pose data, current object posedata, and predicted object pose data may be associated spatiotemporallywith the number of feature points received from the computing platform106A. In particular embodiments, while the wireless network connectionis available, the XR electronic device 102A may then render, on one ormore displays of the XR display device 102A, the first sequence of imageframes by the final re-projection and display functional block 224A. Forexample, the 3D re-projection warping functional block 206A may providethe first sequence of image frames (e.g., 3D images) to the latest IMUfunctional block 218 to associate the first sequence of image frameswith the latest user head pose data and object pose data, for example,and re-project and display the first sequence of image frames on the oneor more displays of the XR display device 102A. In particularembodiments, the wireless network connection may, in some instances, bebecome temporarily unavailable, and thus the first sequence of imageframes may cease being be sent from the computing platform 106A to theXR display device 102A.

FIG. 2B illustrates a detailed embodiment of an extended reality (XR)device with an unavailable network connection for performing a 3Dre-projection warping process once a network connection becomesunavailable, in accordance with presently disclosed embodiments. Inparticular embodiments, once the wireless network connection becometemporarily unavailable (e.g., corresponding to the first sequence ofimage frames no longer being sent from the computing platform 106A tothe XR electronic device 102A), the XR electronic device 102A may thenaccess the number of feature points from a depth map corresponding tothe first sequence of image frames. For example, prior to detecting theinterruption to the image data received from computing platform 106A,the XR electronic device 102A may receive and store the number offeature points corresponding to the first sequence of image frames. Inparticular embodiments, the feature points may be further processed andcurated via the head pose sequence functional block 226A, referenceframe color depths functional block 228A, an object key feature pointand depth sequence extrapolation functional block 230A, and an objectpose sequence extrapolation functional block 232A. In particularembodiments, the XR electronic device 102A may then perform, based onthe number of feature points, a 3D re-warping process by the 3Dre-projection warping functional block 220A to render, by the finalre-projection and display functional block 224A, at least a partialre-rendering of one or more objects included in the first sequence ofimage frames in accordance with movement and position information (e.g.,head pose data, head pose prediction data, object pose data, and objectpose prediction data stored at functional block 214A and the latest userIMU data as provided by the to the latest IMU functional block 218). Forexample, the 3D re-projection warping functional block 220A may utilizethe number of feature points to perform image re-projection warping(e.g., transforming the number of feature points into a 3D object or2.5D object) into at least a partial re-rendering of the one or moreobjects included in the first sequence of image frames (e.g., along withthe latest user head pose data and object pose data). In this way, if awireless network connection becomes temporarily unavailable, a secondsequence of image frames may be rendered for a predetermined period oftime thereafter, or until second image data is received again from thecomputing platform. Thus, the user's XR experience may not ceaseabruptly, and, instead, may gradually and gracefully cease rendering inthe case of network unavailability.

FIG. 2C illustrates another detailed embodiment of an extended reality(XR) system 200C for performing a 2D re-projection warping process, inaccordance with presently disclosed embodiments. The XR system 200C maydiffer from the XR system 200A in that the XR electronic device 102B mayperform a 2D re-warping via a 2D re-warping function 236A and imagedistortion correction via the distortion correction functional block238A. As depicted, the computing platform 106B may include the head posetracking functional block 202B, the rendering engine 204B, the keyfeature point and depth extraction functional block 208B, head posefunctional block 210B, object pose estimation functional block 234, andthe object pose prediction functional block 212B. In particularembodiments, the computing platform 106B may generate image data (e.g.,color frames) corresponding to a first sequence of image frames via therendering engine 204B. The rendering engine 204B may also provide imagedata (e.g., color frames) to the key feature point and depth extractionfunctional block 208B. The key feature point and depth extractionfunctional block 208B may then provide key feature point and depthextraction data to the object pose estimation functional block 234. Theobject pose estimation functional block 234 may be provided to estimateobject poses from the key feature point and depth extraction data. Theobject pose prediction functional block 212B may then receive theestimated object poses and generate object pose prediction data basedthereon.

As further depicted, when the wireless network connection is available,the computing platform 106B may store the number of feature points andthe object pose prediction data to the object key feature points anddepth map data storage functional block 214C of the XR electronic device102B. Similarly, the computing platform 106B may store head pose dataand head pose prediction data to the storage functional block 214B ofthe XR electronic device 102B. As further depicted, while the wirelessnetwork connection remains available, the number of feature points andthe head pose prediction data and the object pose prediction data, aswell as the image data (e.g., color frames) from the rendering engine204B may all be provided to the 3D re-projection warping functionalblock 220B. In particular embodiments, the 3D re-projection warpingfunctional block 220B may then provide output rendering data to the 2Dwarping functional block 236A. Following a 2D warping of the rendingdata and distortion correction 238A (e.g., correction of colordistortion), the color frames corresponding to a first sequence of imageframes may be provided to the display 224B for rendering to a user.

FIG. 2D illustrates another detailed embodiment of an extended reality(XR) system 200D for performing a 2D re-projection warping process withan unavailable network connection, in accordance with presentlydisclosed embodiments. For example, when the wireless network connectionbecomes unavailable, the XR electronic device 102B may receive head posesequence data from the head pose extrapolation functional block 226B,reference frame color and depth data from the reference frame color anddepth functional block 228B, key feature point and depth sequence datafrom the key feature point and depth functional block 230B, and objectpose sequence data from the object pose extrapolation functional block232B. As further depicted, while the wireless network connection remainsunavailable, the head pose sequence data, the reference frame color anddepth data, the key feature point and depth sequence data, and theobject pose sequence data may all be provided to the 3D re-projectionwarping functional block 220B. In particular embodiments, the 3Dre-projection warping functional block 220B may then provide outputrendering data to the 2D warping functional block 236B. Following a 2Dwarping of the rending data and distortion correction 238 (e.g.,correction of color distortion), the color frames corresponding to asecond sequence of image frames may be provided to the display 224B forrendering to a user. In this way, if a wireless network connectionbecomes temporarily unavailable, a second sequence of image frames maybe rendered for a predetermined period of time thereafter, or untilsecond image data is received again from the computing platform. Thus,the user's XR experience may not cease abruptly, and, instead, maygradually and gracefully cease rendering in the case of networkunavailability.

FIG. 3 illustrates is a flow diagram of a method 300 for re-projectingdepth maps on user electronic devices. The method 300 may be performedutilizing one or more processing devices (e.g., XR electronic device102) that may include hardware (e.g., a general purpose processor, agraphic processing unit (GPU), an application-specific integratedcircuit (ASIC), a system-on-chip (SoC), a microcontroller, afield-programmable gate array (FPGA), a central processing unit (CPU),an application processor (AP), a visual processing unit (VPU), a neuralprocessing unit (NPU), a neural decision processor (NDP), or any otherprocessing device(s) that may be suitable for processing image data),software (e.g., instructions running/executing on one or moreprocessors), firmware (e.g., microcode), or some combination thereof.

The method 300 may begin block 302 with the one or more processingdevices (e.g., XR electronic device 102) rendering, on one or moredisplays of an XR electronic device, a first sequence of image framesbased on image data received from an external electronic deviceassociated with the XR electronic device. The method 300 may thencontinue at block 304 with the one or more processing devices (e.g., XRelectronic device 102) detecting an interruption to the image datareceived from the external electronic device associated with the XRdisplay device. In particular embodiments, the XR electronic device andthe external electronic device may be communicatively connected to eachother via a wireless connection, in which the interruption is aninterruption to the wireless connection. The method 300 may thencontinue at block 306 with the one or more processing devices (e.g., XRelectronic device 102) accessing a number of feature points from a depthmap corresponding to the first sequence of image frames, in which thenumber of feature points includes movement and position information ofone or more objects within the first sequence of image frames.

In particular embodiments, prior to detecting the interruption to theimage data received from the external electronic device, the XRelectronic device may receive the number of feature points correspondingto the first sequence of image frames from the external electronicdevice as a background process. In particular embodiments, the XRelectronic device may then store, to a memory of the XR electronicdevice, the number of feature points corresponding to the first sequenceof image frames. The method 300 may then continue at block 308 with theone or more processing devices (e.g., XR electronic device 102)performing a re-warping to at least partially re-render the one or moreobjects based at least in part on the plurality of feature points andspatiotemporal data. In particular embodiments, the XR electronic devicemay access current head pose data and predicted head pose data, in whichthe current head pose data and the predicted head pose data may beassociated with the number of feature points. In particular embodiments,the XR electronic device may also access current object pose data andpredicted object pose data, in which the current object pose data andthe predicted object pose data may be associated with the number offeature points.

In particular embodiments, the XR electronic device may perform there-warping process by determining one or more current color framescorresponding to the first sequence of image frames, and generating,based on the one or more current color frames, one or more updated colorframes corresponding to the first sequence of image frames. The method300 may then conclude at block 310 with the one or more processingdevices (e.g., XR electronic device 102) rendering, on the one or moredisplays of the XR electronic device, a second sequence of image framescorresponding to the partial re-rendering of the one or more objects. Inthis way, if a wireless network connection becomes temporarilyunavailable, the second sequence of image frames may be rendered for apredetermined period of time thereafter, or until second image data isreceived again from the computing platform. Thus, the user's XRexperience may not cease abruptly, and, instead, may gradually andgracefully cease rendering in the case of network unavailability.

FIG. 4 illustrates is a flow diagram of a method 400 for providing depthmap feature points for re-projecting depth maps on user electronicdevices, in accordance with the presently disclosed embodiments. Themethod 400 may be performed utilizing one or more processing devices(e.g., computing platform 106) that may include hardware (e.g., ageneral purpose processor, a graphic processing unit (GPU), anapplication-specific integrated circuit (ASIC), a system-on-chip (SoC),a microcontroller, a field-programmable gate array (FPGA), a centralprocessing unit (CPU), an application processor (AP), a visualprocessing unit (VPU), a neural processing unit (NPU), a neural decisionprocessor (NDP), or any other processing device(s) that may be suitablefor processing image data), software (e.g., instructionsrunning/executing on one or more processors), firmware (e.g.,microcode), or some combination thereof.

The method 400 may begin block 402 with the one or more processingdevices (e.g., computing platform 106) generating image datacorresponding to a first sequence of image frames. The method 400 maythen continue at block 404 with the one or more processing devices(e.g., computing platform 106) accessing a depth map corresponding tothe first sequence of image frames. In particular embodiments, the depthmap may include depth information for one or more most recent imageframes of the first sequence of image frames. The method 400 may thencontinue at block 406 with the one or more processing devices (e.g.,computing platform 106) determining a number of feature points from thedepth map corresponding to the first sequence of image frames based atleast in part on a parametric data reduction (PDR) process, in which thenumber of feature points includes movement and position information ofone or more objects within the first sequence of image frames.

In particular embodiments, the computing platform may determine thenumber of feature points from the depth map may by selecting a subset offeature points of a total set of feature points included in the depthmap. In particular embodiments, the computing platform may alsodetermine the number of feature points from the depth map by determininga plurality of feature points within a predetermined viewing area. Inparticular embodiments, the computing platform may determine the numberof feature points within the predetermined viewing area by determining anumber of feature points within a predefined fovea display area. Inparticular embodiments, the number of feature points within thepredefined fovea display area may include a grouping of feature pointsbased at least part on a nearest-neighbor interpolation. In particularembodiments, the grouping of feature points may include a subgrouping offeature points grouped based at least part on a depth calculation. Inparticular embodiments, the computing platform may determine the numberof feature points by determining a pixel region corresponding to the oneor more objects within the first sequence of image frames, and dividingthe pixel region corresponding to the one or more objects into N pixelsubregions. In particular embodiments, the computing platform may thenextract a number of feature points from the N pixel subregions, in whicheach of the number of feature points is extracted from a respective oneof the N pixel subregions based on a confidence threshold. In particularembodiments, the computing platform may determine a position and anoptical flow for each of the plurality of feature points.

The method 400 may then conclude at block 408 with the one or moreprocessing devices (e.g., computing platform 106) sending the image dataand the number of feature points to an XR electronic device that isexternal to the electronic device. In particular embodiments, thecomputing platform and the XR display device may be communicativelyconnected to each other via a wireless connection. In particularembodiments, the computing platform may also provide current head posedata and predicted head pose data to the XR electronic device, in whichthe current head pose data and the predicted head pose data may beassociated with the number of feature points. In particular embodiment,the computing platform may also provide current object pose data andpredicted object pose data to the XR electronic device, in which thecurrent object pose data and the predicted object pose data areassociated with the number of feature points. In this way, if a wirelessnetwork connection becomes temporarily unavailable, the second sequenceof image frames may be rendered for a predetermined period of timethereafter, or until second image data is received again from thecomputing platform. Thus, the user's XR experience may not ceaseabruptly, and, instead, may gradually and gracefully cease rendering inthe case of network unavailability.

FIGS. 5A and 5B illustrate workflow diagram 500A for determining one ormore current color frames and a frame extrapolation diagram 500B fordetermining a number of image frames to be extrapolated, respectively,in accordance with the presently disclosed embodiments. In particularembodiments, the workflow diagram 500A may be performed, for example, bythe XR electronic device 102. As depicted, the workflow diagram 500A maycommence at block 502 with the XR electronic device 102 obtainingpredicted head poses and object poses. For example, in particularembodiments, the XR electronic device 102 may access current head posedata, predicted head pose data, current object pose data, and predictedobject pose data that may be associated with the number of featurepoints received from the computing platform 106. In particularembodiments, the workflow diagram 500A may then continue at block 504with the XR electronic device 102 re-projecting the current color imageframes. As one example, in particular embodiments, the re-projection ofthe current color image frames may be expressed as vectors u₂ and v₂, asset forth below:

$\begin{matrix}{{u_{2} = \frac{w_{11} + {w_{12}u_{1}} + {w_{13}v_{1}} + {w_{14}{\delta\left( {u_{1},v_{1}} \right)}}}{w_{31} + {w_{32}u_{1}} + {w_{33}v_{1}} + {w_{34}{\delta\left( {u_{1},v_{1}} \right)}}}},} & \left( {{Equation}\mspace{14mu} 1} \right) \\{v_{2} = {\frac{w_{21} + {w_{22}u_{1}} + {w_{23}v_{1}} + {w_{24}{\delta\left( {u_{1},v_{1}} \right)}}}{w_{31} + {w_{32}u_{1}} + {w_{33}v_{1}} + {w_{34}{\delta\left( {u_{1},v_{1}} \right)}}}.}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

In particular embodiments, the workflow diagram 500A may then concludeat block 506 with the XR electronic device 102 obtaining updated colorimage frames. For example, in particular embodiments, the XR electronicdevice 102 may, for example, update a current frame by the 3Dre-projection warping functional block 220 (e.g., 3D re-projectionwarping process) to create a new color frame, which may be utilized, forexample, during one or more delays between head poses and/or objectposes or changes thereto. In particular embodiments, the new color framemay be correlated with changes to head poses and/or object poses basedon, for example, a 2D rotation and translation.

In particular embodiments, when the wireless network becomes temporarilyunavailable, the XR electronic device 102 may utilize the 3Dre-projection warping functional block 220 (e.g., 3D re-projectionwarping process) to create new image frame sequence based on, forexample, the number of feature points corresponding to the firstsequence of image frames and received from the computing platform 106,and the current head pose data, predicted head pose data, current objectpose data, and predicted object pose data that may be associated withthe number of feature points received from the computing platform 106.FIG. 5B illustrates a manner in which a sequence of image frames 508,510, 512, 514, and 516 (e.g., color image frames) may be extrapolatedbased on, for example, one or more of the most recent image frames ofthe sequence of image frames 508, 510, 512, 514, and 516, as wellindicating the determined number of image frames to be extrapolatedbased on, for example, the most recent 3 image frames for 3Dextrapolation and the most recent 2 image frames for 2D extrapolation toperform gradual and graceful ceasing of image rendering in the case ofnetwork unavailability. Specifically, in particular embodiments, the 3most recent image frames of the sequence of image frames 508, 510, 512,514, and 516 may be extrapolated for rendering 3D image frames (e.g.,depth maps), and the 2 most recent image frames of the sequence of imageframes 508, 510, 512, 514, and 516 may be extrapolated for rendering 2Dimage frames. In particular embodiments, the number of most recent imageframes of the sequence of image frames 508, 510, 512, 514, and 516 mayalso depend on, for example, factors including frame frequency andduration, scene complexity, distances between objects, and so forth.

FIGS. 6A, 6B, and 6C illustrate a workflow diagram 600A for selectingkey feature points based on a parametric data reduction (PDR) process, aworkflow diagram 600B for determining and reducing feature points, and aworkflow diagram 600C for determining key point and depth sequenceextrapolation, respectively, in accordance with the presently disclosedembodiments. In particular embodiments, the workflow diagrams 600A,600B, and 600C may each be performed, for example, by the key featurepoint and depth extraction functional block 208A, 208B of the computingplatform 106A, 106B. In particular embodiments, the workflow diagram600A may be provided to reduce feature points selected from a number offrames. In particular embodiments, the workflow diagram 600A maycommence at block 602 with the computing platform 106 determining a newimage frame, a display refresh, or fovea image update. The workflowdiagram 600A may then at block 604 with the computing platform 106subsampling the depth map corresponding the first sequence of images.The workflow diagram 600A may then continue at block 606 with thecomputing platform 106 identifying feature points within a predeterminedviewing area. The workflow diagram 600A may then continue at block 608with the computing platform 106 reducing the feature points within theviewing area based on a predefined fovea display area. The workflowdiagram 600A may then continue at block 610 with the computing platform106 grouping the feature points based on a on a nearest-neighborinterpolation. The workflow diagram 600A may then continue at block 612with the computing platform 106 grouping the feature points based on adepth calculation. The workflow diagram 600A may then continue at block614 with the computing platform 106 running a pixel regional subset ifthe reduced feature points are determined to be greater than, forexample, a predetermined preset value. The workflow diagram 600A maythen conclude at block 616 with the computing platform 106 storing thereduced dataset of feature points (e.g., to be provided to the XRelectronic device 102).

In particular embodiments, the workflow diagram 600B may be provided todetermine feature points selected from a number of frames 618. Forexample, the workflow diagram 600B may be provided to determine one ormore pixel regions corresponding to one or more objects within, forexample, the first sequence of image frames being provided to the XRelectronic device 102 for rendering while the wireless networkconnection is available. The workflow diagram 600B may commence at block620 with the computing platform 106A, 106B dividing the one or morepixel regions corresponding to the one or more objects into N pixelsubregions (e.g., 6 pixel subregions as illustrated in FIG. 6B). Theworkflow diagram 600B may then continue at block 622 with the computingplatform 106A, 106B extracting a number of feature points 626A, 626B,626C, 626D, 626E, and 626F from the N pixel subregions, in which each ofthe number of feature points 626A, 626B, 626C, 626D, 626E, and 626F maybe extracted from a respective one of the N pixel subregions based on aconfidence threshold. The workflow diagram 600B may then continue atblock 624 with the computing platform 106A, 106B selecting the number offeature points 626A, 626B, 626C, 626D, 626E, and 626F from the N pixelsubregions corresponding to the confidence threshold for each of the Npixel subregions. In an example embodiment, the number of feature points626A, 626B, 626C, 626D, 626E, and 626F may be expressed in 3D space as:

$\begin{matrix}{P = {\left\lbrack {x,y,z,\frac{\delta\; x}{\delta\; t},\frac{\delta\; y}{\delta\; t},\frac{\delta\; z}{\delta\; t}} \right\rbrack{\left\{ {P_{0},P_{1},P_{2},P_{3},P_{4},P_{5}} \right\}.}}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

The workflow diagram 600B may then conclude at block 628 with thecomputing platform 106A, 106B determining a position and an optical flowfor each of the number of feature points 626A, 626B, 626C, 626D, 626E,and 626F. The workflow diagram 600C may commence at block 630 with thecomputing platform 106A, 106B getting key point differences fromprevious frames. The workflow diagram 600C may continue at block 632with the computing platform 106A, 106B computing 2D and 3D extrapolationfor derived frames. In an example embodiment, the 2D extrapolation ofderived frames may be expressed as:

$\begin{matrix}{y = {y_{1} + {\frac{x - x_{1}}{x_{2} - x_{1}}{\left( {y_{2} - y_{1}} \right).}}}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

In an example embodiment, the 3D extrapolation of derived frames may beexpressed as:x ₄ =x ₁ +A(x ₂ −x ₁)+B(x ₃ −x ₁)y ₄ =y ₁ +A(y ₂ −y ₁)+B(y ₃ −y ₁)z ₄ =z ₁ +A(z ₂ −z ₁)+B(z ₃ −z ₁)  Equation (5).

The workflow diagram 600C may conclude at block 634 with the computingplatform 106A, 106B obtaining key point sequences for derived frames.

FIGS. 7A, 7B, and 7C illustrate a workflow diagrams 700A, 700B, and 700Cfor determining and estimating head poses and object poses in the mostrecent image frames and/or derived image frames, respectively, inaccordance with the presently disclosed embodiments. In particularembodiments, the workflow diagram 700A may be performed, for example, bythe head pose sequence functional block 226A, 226B of the XR electronicdevice 102A, 102B. In particular embodiments, the workflow diagram 700Amay commence at block 702 with the XR electronic device 102A, 102Bobtaining stored head poses of previous two frames. The workflow diagram700A may continue at block 704 with the XR electronic device 102A, 102Bextrapolating 3D head pose sequence for derived image frames. Theworkflow diagram 700A may continue at block 706 with the XR electronicdevice 102A, 102B obtaining head poses sequence for derived imageframes.

In particular embodiments, the workflow diagram 700B may be performed,for example, by the object pose prediction functional block 212A, 212Bof the computing platform 106A, 106B. In particular embodiments, theworkflow diagram 700B may commence at block 708 with the computingplatform 106 obtaining estimated object poses of previous two frames.The workflow diagram 700A may continue at block 710 with the computingplatform 106A, 106B extrapolating poses of all of objects in the nextframe. The workflow diagram 700A may continue at block 712 with thecomputing platform 106A, 106B obtaining 3D poses of all objects in thenext frame. In particular embodiments, the workflow diagram 700C may beperformed, for example, by the object pose sequence extrapolationfunctional block 232A, 232B of the XR electronic device 102A, 102B. Inparticular embodiments, the workflow diagram 700C may commence at block714 with the XR electronic device 102A, 102B obtaining key featurepoints sequences of derived image frames. The workflow diagram 700C maycontinue at block 716 with the XR electronic device 102A, 102Bestimating poses for each object in each derived image frame. Theworkflow diagram 700A may continue at block 718 with the XR electronicdevice 102A, 102B obtaining object poses sequences of all derived imageframes.

FIGS. 7D and 7E illustrate a workflow diagrams 700D and 700E fordetermining object pose estimation and performing a 2D image warping andre-projection, respectively, in accordance with the presently disclosedembodiments. In particular embodiments, the workflow diagram 700D may beperformed, for example, by the object pose estimation functional block234 of the computing platform 106B. In particular embodiments, theworkflow diagram 700D may commence at block 720 with the computingplatform 106B determining corresponding 3D feature point sets A in areference image frame and 3D feature point sets B in a current imageframe. The workflow diagram 700D may continue at block 722 with thecomputing platform 106B computing an error function with respect to thecurrent frame and the reference image frame. In an example embodiment,the error function may be expressed as:

$\begin{matrix}\begin{matrix}{{{RA} + t} = B} \\{{err} = {\sum\limits_{i = 1}^{N}{{{{RA}^{l} + t - B}}^{2}.}}}\end{matrix} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

In the Equation (6), R may represent rotation, in particularembodiments. The workflow diagram 700D may continue at block 724 withthe computing platform 106B computing one or more centroid values withrespect to 3D feature point sets A in the reference image frame and 3Dfeature point sets B in the current image frame. In an exampleembodiment, the one or more centroid values may be expressed as:

$\begin{matrix}{{{centroid}_{A} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}A^{i}}}},{{centroid}_{B} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{B^{i}.}}}}} & \left( {{Equation}\mspace{14mu} 7} \right)\end{matrix}$

The workflow diagram 700D may continue at block 726 with the computingplatform 106B creating a criterion function based on the one or morecentroid values. In an example embodiment, the criterion function H maybe expressed as:H=(A−centroid_(A))(B−centroid_(B))^(T)  Equation (8).

The workflow diagram 700D may continue at block 728 with the computingplatform 106B performing a singular value decomposition (SVD) withrespect to the reference frame and the current image frame. In anexample embodiment, the criterion function may be expressed as:[U,S,V]=SVD(H)  Equation (9).

In the Equation (9), U, S, and V may each represent, for example, afactorized matrix. The workflow diagram 700D may continue at block 730with the computing platform 106B computing a rotation matrix based onthe SVD decomposition. In an example embodiment, the rotation matrix Rmay be expressed as:R=VU ^(T)  Equation (10).

In the Equation (10), each of the factorized matrices U and V may betransformed by a transformation matrix T The workflow diagram 700D maythen conclude at block 732 with the computing platform 106B computing atranslation matrix based on the rotation of the SVD decomposition. In anexample embodiment, the translation matrix may be expressed as:RA+t=BR·centroid_(A) +t=centroid_(B)t=centroid_(B) −R·centroid_(A)  Equation (11).

In particular embodiments, the workflow diagram 700E may be performed,for example, by the 2D re-warping function 236A, 236B of the XRelectronic device 102A, 102B. In particular embodiments, the workflowdiagram 700E may commence at block 734 with the XR electronic device102A, 102B obtaining predicted head poses and object poses. The workflowdiagram 700E may continue at block 736 with the XR electronic device102A, 102B warping a current color image frame utilizing 2D rotation andtranslation. The workflow diagram 700E may then conclude at block 738with the XR electronic device 102A, 102B obtaining updated color imageframes.

FIG. 8 illustrates an example computer system 800 that may be utilizedfor re-projecting depth maps on user electronic devices, in accordancewith the presently disclosed embodiments. In particular embodiments, oneor more computer systems 800 perform one or more steps of one or moremethods described or illustrated herein. In particular embodiments, oneor more computer systems 800 provide functionality described orillustrated herein. In particular embodiments, software running on oneor more computer systems 800 performs one or more steps of one or moremethods described or illustrated herein or provides functionalitydescribed or illustrated herein. Particular embodiments include one ormore portions of one or more computer systems 800. Herein, reference toa computer system may encompass a computing device, and vice versa,where appropriate. Moreover, reference to a computer system mayencompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems800. This disclosure contemplates computer system 800 taking anysuitable physical form. As example and not by way of limitation,computer system 800 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (e.g., a computer-on-module(COM) or system-on-module (SOM)), a desktop computer system, a laptop ornotebook computer system, an interactive kiosk, a mainframe, a mesh ofcomputer systems, a mobile telephone, a personal digital assistant(PDA), a server, a tablet computer system, an augmented/virtual realitydevice, or a combination of two or more of these. Where appropriate,computer system 800 may include one or more computer systems 800; beunitary or distributed; span multiple locations; span multiple machines;span multiple data centers; or reside in a cloud, which may include oneor more cloud components in one or more networks.

Where appropriate, one or more computer systems 800 may perform withoutsubstantial spatial or temporal limitation one or more steps of one ormore methods described or illustrated herein. As an example, and not byway of limitation, one or more computer systems 800 may perform in realtime or in batch mode one or more steps of one or more methods describedor illustrated herein. One or more computer systems 800 may perform atdifferent times or at different locations one or more steps of one ormore methods described or illustrated herein, where appropriate.

In particular embodiments, computer system 800 includes a processor 802,memory 804, storage 806, an input/output (I/O) interface 808, acommunication interface 810, and a bus 812. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 802 includes hardware for executinginstructions, such as those making up a computer program. As an example,and not by way of limitation, to execute instructions, processor 802 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 804, or storage 806; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 804, or storage 806. In particular embodiments, processor802 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 802 including anysuitable number of any suitable internal caches, where appropriate. Asan example, and not by way of limitation, processor 802 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 804 or storage 806, andthe instruction caches may speed up retrieval of those instructions byprocessor 802.

Data in the data caches may be copies of data in memory 804 or storage806 for instructions executing at processor 802 to operate on; theresults of previous instructions executed at processor 802 for access bysubsequent instructions executing at processor 802 or for writing tomemory 804 or storage 806; or other suitable data. The data caches mayspeed up read or write operations by processor 802. The TLBs may speedup virtual-address translation for processor 802. In particularembodiments, processor 802 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 802 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 802 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 802. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 804 includes main memory for storinginstructions for processor 802 to execute or data for processor 802 tooperate on. As an example, and not by way of limitation, computer system800 may load instructions from storage 806 or another source (such as,for example, another computer system 800) to memory 804. Processor 802may then load the instructions from memory 804 to an internal registeror internal cache. To execute the instructions, processor 802 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 802 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor802 may then write one or more of those results to memory 804. Inparticular embodiments, processor 802 executes only instructions in oneor more internal registers or internal caches or in memory 804 (asopposed to storage 806 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 804 (as opposedto storage 806 or elsewhere).

One or more memory buses (which may each include an address bus and adata bus) may couple processor 802 to memory 804. Bus 812 may includeone or more memory buses, as described below. In particular embodiments,one or more memory management units (MMUs) reside between processor 802and memory 804 and facilitate accesses to memory 804 requested byprocessor 802. In particular embodiments, memory 804 includes randomaccess memory (RAM). This RAM may be volatile memory, where appropriate.Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM(SRAM). Moreover, where appropriate, this RAM may be single-ported ormulti-ported RAM. This disclosure contemplates any suitable RAM. Memory804 may include one or more memories 804, where appropriate. Althoughthis disclosure describes and illustrates particular memory, thisdisclosure contemplates any suitable memory.

In particular embodiments, storage 806 includes mass storage for data orinstructions. As an example, and not by way of limitation, storage 806may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage806 may include removable or non-removable (or fixed) media, whereappropriate. Storage 806 may be internal or external to computer system800, where appropriate. In particular embodiments, storage 806 isnon-volatile, solid-state memory. In particular embodiments, storage 806includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 806 taking any suitable physicalform. Storage 806 may include one or more storage control unitsfacilitating communication between processor 802 and storage 806, whereappropriate. Where appropriate, storage 806 may include one or morestorages 806. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 808 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 800 and one or more I/O devices. Computer system800 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 800. As an example, and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 806 for them. Where appropriate, I/O interface 808 mayinclude one or more device or software drivers enabling processor 802 todrive one or more of these I/O devices. I/O interface 808 may includeone or more I/O interfaces 806, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 810 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 800 and one or more other computer systems 800 or one ormore networks. As an example, and not by way of limitation,communication interface 810 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI network. Thisdisclosure contemplates any suitable network and any suitablecommunication interface 810 for it.

As an example, and not by way of limitation, computer system 800 maycommunicate with an ad hoc network, a personal area network (PAN), alocal area network (LAN), a wide area network (WAN), a metropolitan areanetwork (MAN), or one or more portions of the Internet or a combinationof two or more of these. One or more portions of one or more of thesenetworks may be wired or wireless. As an example, computer system 800may communicate with a wireless PAN (WPAN) (such as, for example, aBLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephonenetwork (such as, for example, a Global System for Mobile Communications(GSM) network), or other suitable wireless network or a combination oftwo or more of these. Computer system 800 may include any suitablecommunication interface 810 for any of these networks, whereappropriate. Communication interface 810 may include one or morecommunication interfaces 810, where appropriate. Although thisdisclosure describes and illustrates a particular communicationinterface, this disclosure contemplates any suitable communicationinterface.

In particular embodiments, bus 812 includes hardware, software, or bothcoupling components of computer system 800 to each other. As an example,and not by way of limitation, bus 812 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 812may include one or more buses 812, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

Herein, “automatically” and its derivatives means “without humanintervention,” unless expressly indicated otherwise or indicatedotherwise by context.

The embodiments disclosed herein are only examples, and the scope ofthis disclosure is not limited to them. Embodiments according to theinvention are in particular disclosed in the attached claims directed toa method, a storage medium, a system and a computer program product,wherein any feature mentioned in one claim category, e.g. method, may beclaimed in another claim category, e.g. system, as well. Thedependencies or references back in the attached claims are chosen forformal reasons only. However, any subject matter resulting from adeliberate reference back to any previous claims (in particular multipledependencies) may be claimed as well, so that any combination of claimsand the features thereof are disclosed and may be claimed regardless ofthe dependencies chosen in the attached claims. The subject-matter whichmay be claimed comprises not only the combinations of features as setout in the attached claims but also any other combination of features inthe claims, wherein each feature mentioned in the claims may be combinedwith any other feature or combination of other features in the claims.Furthermore, any of the embodiments and features described or depictedherein may be claimed in a separate claim and/or in any combination withany embodiment or feature described or depicted herein or with any ofthe features of the attached claims.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. A method comprising, by an extended reality (XR)display device: rendering, on one or more displays of the XR displaydevice, a first sequence of image frames based on image data receivedfrom an external electronic device associated with the XR displaydevice; detecting an interruption to the image data received from theexternal electronic device associated with the XR display device;accessing a plurality of feature points from a depth map correspondingto the first sequence of image frames, wherein the plurality of featurepoints comprises movement and position information of one or moreobjects within the first sequence of image frames; performing are-warping to at least partially re-render the one or more objects basedat least in part on the plurality of feature points and spatiotemporaldata; and rendering, on the one or more displays of the XR displaydevice, a second sequence of image frames corresponding to the partialre-rendering of the one or more objects.
 2. The method of claim 1,wherein the spatiotemporal data comprises one or more of: current headpose data and predicted head pose data; or current object pose data andpredicted object pose data.
 3. The method of claim 1, wherein performingthe re-warping comprises: determining one or more current color framescorresponding at least partially to the first sequence of image frames;and generating, based on the one or more current color frames, one ormore updated color frames corresponding at least partially to the firstsequence of image frames.
 4. The method of claim 1, further comprising:prior to detecting the interruption to the image data received from theexternal electronic device: receiving the plurality of feature pointscorresponding to the first sequence of image frames from the externalelectronic device as a background process; and storing, to a memory ofthe XR display device, the plurality of feature points corresponding tothe first sequence of image frames.
 5. The method of claim 1, whereinthe second sequence of image frames is rendered: for a predeterminedperiod of time; or until second image data is received from the externalelectronic device.
 6. An extended reality (XR) display devicecomprising: a transceiver; one or more displays; one or morenon-transitory computer-readable storage media including instructions;and one or more processors coupled to the storage media, the one or moreprocessors configured to execute the instructions to: render, on one ormore displays of the XR display device, a first sequence of image framesbased on image data received from an external electronic deviceassociated with the XR display device; detect an interruption to theimage data received from the external electronic device associated withthe XR display device; access a plurality of feature points from a depthmap corresponding to the first sequence of image frames, wherein theplurality of feature points comprises movement and position informationof one or more objects within the first sequence of image frames;perform a re-warping to at least partially re-render the one or moreobjects based at least in part on the plurality of feature points andspatiotemporal data; and render, on the one or more displays of the XRdisplay device, a second sequence of image frames corresponding to thepartial re-rendering of the one or more objects.
 7. The XR displaydevice of claim 6, wherein the spatiotemporal data comprises one or moreof: current head pose data and predicted head pose data; or currentobject pose data and predicted object pose data.
 8. The XR displaydevice of claim 6, wherein the instructions to perform the re-warpingfurther comprise instructions to: determine one or more current colorframes corresponding at least partially to the first sequence of imageframes; and generate, based on the one or more current color frames, oneor more updated color frames corresponding at least partially to thefirst sequence of image frames.
 9. The XR display device of claim 6, theone or more processors are further configured to execute theinstructions to: prior to detecting the interruption to the image datareceived from the external electronic device: receive the plurality offeature points corresponding to the first sequence of image frames fromthe external electronic device as a background process; and store, to amemory of the XR display device, the plurality of feature pointscorresponding to the first sequence of image frames.
 10. The XR displaydevice of claim 6, wherein the second sequence of image frames isrendered: for a predetermined period of time; or until second image datais received from the external electronic device.
 11. A non-transitorycomputer-readable medium comprising instructions that, when executed byone or more processors of an extended reality (XR) display device, causethe one or more processors to: render, on one or more displays of the XRdisplay device, a first sequence of image frames based on image datareceived from an external electronic device associated with the XRdisplay device; detect an interruption to the image data received fromthe external electronic device associated with the XR display device;access a plurality of feature points from a depth map corresponding tothe first sequence of image frames, wherein the plurality of featurepoints comprises movement and position information of one or moreobjects within the first sequence of image frames; perform a re-warpingto at least partially re-render the one or more objects based at leastin part on the plurality of feature points and spatiotemporal data; andrender, on the one or more displays of the XR display device, a secondsequence of image frames corresponding to the partial re-rendering ofthe one or more objects.
 12. The non-transitory computer-readable mediumof claim 11, wherein the spatiotemporal data comprises one or more of:current head pose data and predicted head pose data; or current objectpose data and predicted object pose data.
 13. The non-transitorycomputer-readable medium of claim 11, wherein the instructions toperform the re-warping further comprise instructions to: determine oneor more current color frames corresponding at least partially to thefirst sequence of image frames; and generate, based on the one or morecurrent color frames, one or more updated color frames corresponding atleast partially to the first sequence of image frames.
 14. Thenon-transitory computer-readable medium of claim 11, further comprisinginstructions that, when executed by the one or more processors of XRdisplay device, cause the one or more processors to: prior to detectingthe interruption to the image data received from the external electronicdevice: receive the plurality of feature points corresponding to thefirst sequence of image frames from the external electronic device as abackground process; and store, to a memory of the XR display device, theplurality of feature points corresponding to the first sequence of imageframes.
 15. The non-transitory computer-readable medium of claim 11,wherein the second sequence of image frames is rendered: for apredetermined period of time; or until second image data is receivedfrom the external electronic device.
 16. A method comprising, by anelectronic device: generating image data corresponding to a firstsequence of image frames; accessing a depth map corresponding to thefirst sequence of image frames; determining a plurality of featurepoints from the depth map corresponding to the first sequence of imageframes based at least in part on parametric data reduction (PDR),wherein the plurality of feature points comprises movement and positioninformation of one or more objects within the first sequence of imageframes; and sending the image data and the plurality of feature pointsto an extended reality (XR) display device that is external to theelectronic device.
 17. The method of claim 16, wherein the depth mapcomprises depth information for one or more most recent image frames ofthe first sequence of image frames.
 18. The method of claim 16, whereindetermining the plurality of feature points from the depth map comprisesselecting a subset of feature points of a total set of feature pointsincluded in the depth map.
 19. The method of claim 16, whereindetermining the plurality of feature points from the depth map comprisesdetermining a plurality of feature points within a predetermined viewingarea.
 20. The method of claim 19, wherein determining the plurality offeature points within the predetermined viewing area comprisesdetermining a plurality of feature points within a predefined foveadisplay area, wherein the plurality of feature points within thepredefined fovea display area comprises a grouping of feature pointsbased at least part on a nearest-neighbor interpolation, and wherein thegrouping of feature points comprises a subgrouping of feature pointsgrouped based at least part on a depth calculation.
 21. An electronicdevice comprising: a transceiver; one or more non-transitorycomputer-readable storage media including instructions; and one or moreprocessors coupled to the storage media, the one or more processorsconfigured to execute the instructions to: generate image datacorresponding to a first sequence of image frames; access a depth mapcorresponding to the first sequence of image frames; determine aplurality of feature points from the depth map corresponding to thefirst sequence of image frames based at least in part on parametric datareduction (PDR), wherein the plurality of feature points comprisesmovement and position information of one or more objects within thefirst sequence of image frames; and send the image data and theplurality of feature points to an extended reality (XR) display devicethat is external to the electronic device.
 22. The electronic device ofclaim 21, wherein the depth map comprises depth information for one ormore most recent image frames of the first sequence of image frames. 23.The electronic device of claim 21, wherein determining the plurality offeature points from the depth map comprises selecting a subset offeature points of a total set of feature points included in the depthmap.
 24. The electronic device of claim 21, wherein determining theplurality of feature points from the depth map comprises determining aplurality of feature points within a predetermined viewing area.
 25. Theelectronic device of claim 24, wherein the instruction to determine theplurality of feature points within the predetermined viewing areafurther comprise instruction to determine a plurality of feature pointswithin a predefined fovea display area, wherein the plurality of featurepoints within the predefined fovea display area comprises a grouping offeature points based at least part on a nearest-neighbor interpolation,and wherein the grouping of feature points comprises a subgrouping offeature points grouped based at least part on a depth calculation.
 26. Anon-transitory computer-readable medium comprising instructions that,when executed by one or more processors of an electronic device, causethe one or more processors to: generate image data corresponding to afirst sequence of image frames; access a depth map corresponding to thefirst sequence of image frames; determine a plurality of feature pointsfrom the depth map corresponding to the first sequence of image framesbased at least in part on parametric data reduction (PDR), wherein theplurality of feature points comprises movement and position informationof one or more objects within the first sequence of image frames; andsend the image data and the plurality of feature points to an extendedreality (XR) display device that is external to the electronic device.27. The non-transitory computer-readable medium of claim 26, wherein thedepth map comprises depth information for one or more most recent imageframes of the first sequence of image frames.
 28. The non-transitorycomputer-readable medium of claim 26, wherein determining the pluralityof feature points from the depth map comprises selecting a subset offeature points of a total set of feature points included in the depthmap.
 29. The non-transitory computer-readable medium of claim 26,wherein determining the plurality of feature points from the depth mapcomprises determining a plurality of feature points within apredetermined viewing area.
 30. The non-transitory computer-readablemedium of claim 29, wherein the instruction to determine the pluralityof feature points within the predetermined viewing area further compriseinstruction to determine a plurality of feature points within apredefined fovea display area, wherein the plurality of feature pointswithin the predefined fovea display area comprises a grouping of featurepoints based at least part on a nearest-neighbor interpolation, andwherein the grouping of feature points comprises a subgrouping offeature points grouped based at least part on a depth calculation.